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MEMORANDUM FOR: Mr. Tweedy, D/DCI/IC 


SUBJECT: 


Evaluation of Intelligence Products 


20 July 1972 

ts J C 


25X1 A 


1 . These comments and my request for your approval of the 
policy set forth in this memorandum result from consideration of 
NSCIC Working Group activities to date and my reading of the 
report, "CIA Evaluation Systems - Preliminary Observations," 
which the Inspector General made available to me on 19 July. 

The attached 12 June memorandum for the Inspector General from 
Mr. Colby, which you may already have seen, also pertains. 

2. I recommend that the Product Review role with respect to 
the evaluation of intelligence products be as follows: 

a. To participate as appropriate in such reviews of the 
responsiveness of intelligence products to consumer needs 

as may be sponsored by the NSCIC Working Group. It is 
anticipated that these reviews will focus primarily on the 
assessment of intelligence responsiveness in crisis situations 
(as in the case of the India-Pakistan crisis of 1971) and 
on the review of the adequacy of intelligence inputs to specific 
problems of high policy interest (as in the case of inputs to 
NSSM-69) . 

b. To arrange, where appropriate, for consumer evaluation 
of specific intelligence products which are of high consumer 
interest. The proposal to have NSCIC Working Group members 
assume responsibility for completion of evaluation forms on 
NIEs and SNIEs for a period of time is an example. 

c. To identify specific problem areas which appear to call 
for corrective action. These will pertain more to identification 
of gaps in the production program (or, perhaps, "overkill") 

than to the specific evaluation of production output. My pending 
proposal for the application of special effort to the Soviet 
R&D effort is an illustration. 

d. To conduct research in areas identified as posing 
problems related to production. The proposed project with 
OMS/PSS to improve the quality of future production by 
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developing better recruitment criteria for the selection of 
analysts is one example. Another is the proposal that PRG 
look into the impact which delays in the processing of certain 
types of data are having on the production effort. 

3. Your approval of the foregoing would mean that the development ,/ 
of product evaluation systems for use within the production elements p 

of the intelligence community is recognized as an in-house responsibility.'^ 

The program which Mr. Colby and the Inspector General are pursuing 1 

within CIA , as per Mr. Colby's memorandum of 12 June, would be 
recognized as an Agency responsibility, supporting the aims of the 
DCI in response to the Presidential memorandum of 5 November 1971 and 
to the new NSCID No. 1. PRG would have no role in this activity 

except that, once the CIA is operating, I suggest it would be 

appropriate for PRG to draft a letter for the DCI to send to I I 25X1 

I | citing the CIA actions and requesting that they 

advise him as to the internal measures they are taking for quality 
control and evaluation of their products. 


25X1 A 


Vj/iief, PRG/IC 

Attachment’ : 
as stated 
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12 June 1972 


MEMORANDUM FOR: Inspector General 
SUBJECT : Evaluation and Productivity 


1. Thanks to the report on "CIA Evaluation Systems -- Pre- _ 
liminary Observations" submitted by the IG, a useful discussion o 
productivity and evaluations took place at the Director s Annua 
Conference. While there was some discussion, if not dissen , as 
your appraisal of what systems actually exist, there was genera 
agreement that the subject requires more and continuing attention. 
tu rq rector is particularly concerned as to tx,e imeireiauonst p 

betwl g^ivation of S ' 

and thra^Te eTo”ich evaluations can contribute to maintaining 

these at a high level. He c harged the Deputi esjvrth examining the 

actual status of their evaluatmuTyste^^ th^Tdirectorates 

F f iv e y e ars a£ter)> and 

he called for a discussion ofthis subject in detaiTafTT^Scoming 
Deputies Meeting. The evaluation assessment should include 
sorts of post-mortems and assessments of effectiveness, m addition 
• to any specific evaluation systems which may exist. 

2. The Inspector General is directed to gei^aggx^^ 
vision of the assembly, of .this study and the develo^enl^UliS-a.pife 
^^Sd^Ematerial for a future discussion of evaluation 

This should be carried out in close 

coordination with the directorates through whatever offacer is 
by each Deputy as a point of contact for his directorate. The meeting 
agenda should include the following subjects: 


a 


Inventory of current evaluation systems, 


b. Utilization of evaluation systems in program, personnel 
and other decision-making; 


[; 


. — - . i ei - 


■1 3 \ 


j cas-? i 

I V. • ^ * j 3 1 Htc'v'-'C 

y y s d i J L 1 i fi * A " ! t: .nr.i,ri-inj t.i l 
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c. Interrelationships and comparability of evaluation systems; 


d. . C r itigue^gl ^as r-&nt^e. v a 1 ,Ui 1 i-u 
reference to the needs of unit, office /division, directorate and 
Agency levels of management respectively; 

e. Recommendations. 

3. While we' should move with all deliberate speed on this action, 

I think it important that the discussion and resultant recommendations 
be fully informed and responsive to CIA's unique requirements for flexi- 
bility, security, and adjustment to customer and policy needs. Thus, no 
date is established for this .Deputies ' discussion, but you and the Deputies' 
representatives should recommend a date which will give the time needed 
to do a thorough and sensitive study responsive to the Director's desires 
and appropriate for presentation to him. 

I 25X1 A 


Executive Director- Comptroller 


cc: DD/I 
DD/P 
DD/S&T 
DD/S 


G : n S-V7J r 3 r* ■ ! *7 ? ^ 
j q ^ J iJ ^ ^ | i i 
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® 6. Andy also said he was working with DIA where a project 

is underway to produce an experimental product using Bayesian 
techniques. My discussion of the effort underway in PRG emphasized 
my feeling that production managers would have to have considerable 
control over the effort if they were to be receptive to the results, 
but Andy is of the belief that attention also needs to be given to 
a community mechanism outside DDI , DDS&T, and DIA. He has an idea 
that the use of outside consultants and contractors working with key 
elements of the DCI/IC staff might prove feasible. He also said that 
Pat Parker of ASD/I has advised him DIA already is using outside 
consultants to train selected analysts in advanced analytical 
methodologies. 

— _ — — — — — - ~ ~ ~ j 

7. Since Andy had mentioned the need for quality control in 

his 27 July memorandum, I told him I was aware of an effort already 
underway to survey evaluation methods and by close of the year was ! 
charged with preparing a study recommending how an improved evaluatio/i 
system could be brought into effect. Pickett asked if he could be j 

put in touch with this group and I declined on the grounds that the [j 
activity was being closely held and should be so regarded at least j 
until its report had been completed. I did not identify the office J 
which was preparing the study. ! 

8. We also discussed briefly the discussions PRG has had with 
OMS/PSS focused on ways and means of identifying the attributes of 
good and bad analysts. I reported, however, that current indications 
were this would be a rather expensive project and decision had not 
been reached as to whether to proceed with it. 

9. Andy indicated most of his present activities relate to 
formulation of a net assessment program. Commander Robin Pirie, 

who has had experience in OSD/SA prior to recent sea duty, has joined 
Andy's staff and will work on net assessment matters. 

10. With respect to on-going projects, Andy is still reviewing 
notes on the proposed revision of the India-Pakistan study; the 
NSSM-69 study is in second redraft; Bob Baraz of State is heading 
the team on the Arab-Israeli cease-fire study : V I 

of CIA is heading the Jordan/Fedayeen civil war study; and | 
of DIA will do the LAMSON-719 study. | Iwants JCS participation 
in this. 


>5X1A 

125X1 

25X1 


11. Andy's office is being moved next week to Room 302, E.O.B. 
The telephone number will remain the same. 


25X1 A 


Distribution: \ 

orig - PRG subject 
1 - Mr. Twe edy 
1 - I I chrono 
1 - PRG chrono 


ihief, PRG/IC 
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Excerpts from MOR of I 
th Andrew Marshall, NSC Staff 


12 July 1972, Subj: 


Conference 


"4. With respect to the expression of community judgments on 
intentions and objectives, I suggested he might consider proposing 
that during crisis sutations, ONE establish an ad hoc group, including 
community representation, which would function full-time and prepare 
SNIEs as necessary to maintain the currency of community judgments 
as the situation evolved. This would not put ONE in the current 
intelligence business but would provide a means by which policy 
makers could be kept aware of intelligence community judgments 
on major facets of the crisis situation." 

"5. Andy said he did not consider the ONE style of writing was 
adequate for such purposes. In his view, ONE writin g is delibe rately 
fuzzy and does not convey the^cTear and di stihct*expressl on of views 
'heeded by decision makers. 

"11. We discussed the matter of feedback from users to intelligence 
producers, and Andy's view was that his attempts to increase feedback 
had not shown much receptivity at the intelligence end. He said 
the primary reaction has been a 'bristling' or a comment to 'get it 
in writing from Mr. Kissinger' without much else happening." 
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